The multimodality cell segmentation challenge: toward universal solutions.

Cell segmentation is a critical step for quantitative single-cell analysis in microscopy images. Existing cell segmentation methods are often tailored to specific modalities or require manual interventions to specify hyper-parameters in different experimental settings. Here, we present a multimodality cell segmentation benchmark, comprising more than 1,500 labeled images derived from more than 50 diverse biological experiments. The top participants developed a Transformer-based deep-learning algorithm that not only exceeds existing methods but can also be applied to diverse microscopy images across imaging platforms and tissue types without manual parameter adjustments. This benchmark and the improved algorithm offer promising avenues for more accurate and versatile cell analysis in microscopy imaging.

Accurate single-molecule spot detection for image-based spatial transcriptomics with weakly supervised deep learning.

Image-based spatial transcriptomics methods enable transcriptome-scale gene expression measurements with spatial information but require complex, manually-tuned analysis pipelines. We present Polaris, an analysis pipeline for image-based spatial transcriptomics that combines deep learning models for cell segmentation and spot detection with a probabilistic gene decoder to quantify single-cell gene expression accurately. Polaris offers a unifying, turnkey solution for analyzing spatial transcriptomics data from MERFSIH, seqFISH, or ISS experiments. Polaris is available through the DeepCell software library (https://github.com/vanvalenlab/deepcell-spots) and https://www.deepcell.org.

A microwell platform for high-throughput longitudinal phenotyping and selective retrieval of organoids.

Organoids are powerful experimental models for studying the ontogeny and progression of various diseases including cancer. Organoids are conventionally cultured in bulk using an extracellular matrix mimic. However, bulk-cultured organoids physically overlap, making it impossible to track the growth of individual organoids over time in high throughput. Moreover, local spatial variations in bulk matrix properties make it difficult to assess whether observed phenotypic heterogeneity between organoids results from intrinsic cell differences or differences in the microenvironment. Here, we developed a microwell-based method that enables high-throughput quantification of image-based parameters for organoids grown from single cells, which can further be retrieved from their microwells for molecular profiling. Coupled with a deep learning image-processing pipeline, we characterized phenotypic traits including growth rates, cellular movement, and apical-basal polarity in two CRISPR-engineered human gastric organoid models, identifying genomic changes associated with increased growth rate and changes in accessibility and expression correlated with apical-basal polarity. A record of this paper's transparent peer review process is included in the supplemental information.

A spatially resolved timeline of the human maternal-fetal interface.

Beginning in the first trimester, fetally derived extravillous trophoblasts (EVTs) invade the uterus and remodel its spiral arteries, transforming them into large, dilated blood vessels. Several mechanisms have been proposed to explain how EVTs coordinate with the maternal decidua to promote a tissue microenvironment conducive to spiral artery remodelling (SAR)1-3. However, it remains a matter of debate regarding which immune and stromal cells participate in these interactions and how this evolves with respect to gestational age. Here we used a multiomics approach, combining the strengths of spatial proteomics and transcriptomics, to construct a spatiotemporal atlas of the human maternal-fetal interface in the first half of pregnancy. We used multiplexed ion beam imaging by time-of-flight and a 37-plex antibody panel to analyse around 500,000 cells and 588 arteries within intact decidua from 66 individuals between 6 and 20 weeks of gestation, integrating this dataset with co-registered transcriptomics profiles. Gestational age substantially influenced the frequency of maternal immune and stromal cells, with tolerogenic subsets expressing CD206, CD163, TIM-3, galectin-9 and IDO-1 becoming increasingly enriched and colocalized at later time points. By contrast, SAR progression preferentially correlated with EVT invasion and was transcriptionally defined by 78 gene ontology pathways exhibiting distinct monotonic and biphasic trends. Last, we developed an integrated model of SAR whereby invasion is accompanied by the upregulation of pro-angiogenic, immunoregulatory EVT programmes that promote interactions with the vascular endothelium while avoiding the activation of maternal immune cells.

The immunoregulatory landscape of human tuberculosis granulomas.

Tuberculosis (TB) in humans is characterized by formation of immune-rich granulomas in infected tissues, the architecture and composition of which are thought to affect disease outcome. However, our understanding of the spatial relationships that control human granulomas is limited. Here, we used multiplexed ion beam imaging by time of flight (MIBI-TOF) to image 37 proteins in tissues from patients with active TB. We constructed a comprehensive atlas that maps 19 cell subsets across 8 spatial microenvironments. This atlas shows an IFN-γ-depleted microenvironment enriched for TGF-ß, regulatory T cells and IDO1+ PD-L1+ myeloid cells. In a further transcriptomic meta-analysis of peripheral blood from patients with TB, immunoregulatory trends mirror those identified by granuloma imaging. Notably, PD-L1 expression is associated with progression to active TB and treatment response. These data indicate that in TB granulomas, there are local spatially coordinated immunoregulatory programs with systemic manifestations that define active TB.

Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning.

A principal challenge in the analysis of tissue imaging data is cell segmentation-the task of identifying the precise boundary of every cell in an image. To address this problem we constructed TissueNet, a dataset for training segmentation models that contains more than 1 million manually labeled cells, an order of magnitude more than all previously published segmentation training datasets. We used TissueNet to train Mesmer, a deep-learning-enabled segmentation algorithm. We demonstrated that Mesmer is more accurate than previous methods, generalizes to the full diversity of tissue types and imaging platforms in TissueNet, and achieves human-level performance. Mesmer enabled the automated extraction of key cellular features, such as subcellular localization of protein signal, which was challenging with previous approaches. We then adapted Mesmer to harness cell lineage information in highly multiplexed datasets and used this enhanced version to quantify cell morphology changes during human gestation. All code, data and models are released as a community resource.

DeepCell Kiosk: scaling deep learning-enabled cellular image analysis with Kubernetes.

Deep learning is transforming the analysis of biological images, but applying these models to large datasets remains challenging. Here we describe the DeepCell Kiosk, cloud-native software that dynamically scales deep learning workflows to accommodate large imaging datasets. To demonstrate the scalability and affordability of this software, we identified cell nuclei in 106 1-megapixel images in ~5.5 h for ~US$250, with a cost below US$100 achievable depending on cluster configuration. The DeepCell Kiosk can be downloaded at https://github.com/vanvalenlab/kiosk-console ; a persistent deployment is available at https://deepcell.org/ .

Single-cell metabolic profiling of human cytotoxic T cells.

Cellular metabolism regulates immune cell activation, differentiation and effector functions, but current metabolic approaches lack single-cell resolution and simultaneous characterization of cellular phenotype. In this study, we developed an approach to characterize the metabolic regulome of single cells together with their phenotypic identity. The method, termed single-cell metabolic regulome profiling (scMEP), quantifies proteins that regulate metabolic pathway activity using high-dimensional antibody-based technologies. We employed mass cytometry (cytometry by time of flight, CyTOF) to benchmark scMEP against bulk metabolic assays by reconstructing the metabolic remodeling of in vitro-activated naive and memory CD8+ T cells. We applied the approach to clinical samples and identified tissue-restricted, metabolically repressed cytotoxic T cells in human colorectal carcinoma. Combining our method with multiplexed ion beam imaging by time of flight (MIBI-TOF), we uncovered the spatial organization of metabolic programs in human tissues, which indicated exclusion of metabolically repressed immune cells from the tumor-immune boundary. Overall, our approach enables robust approximation of metabolic and functional states in individual cells.

Deep learning for cellular image analysis.

Recent advances in computer vision and machine learning underpin a collection of algorithms with an impressive ability to decipher the content of images. These deep learning algorithms are being applied to biological images and are transforming the analysis and interpretation of imaging data. These advances are positioned to render difficult analyses routine and to enable researchers to carry out new, previously impossible experiments. Here we review the intersection between deep learning and cellular image analysis and provide an overview of both the mathematical mechanics and the programming frameworks of deep learning that are pertinent to life scientists. We survey the field's progress in four key applications: image classification, image segmentation, object tracking, and augmented microscopy. Last, we relay our labs' experience with three key aspects of implementing deep learning in the laboratory: annotating training data, selecting and training a range of neural network architectures, and deploying solutions. We also highlight existing datasets and implementations for each surveyed application.

A Structured Tumor-Immune Microenvironment in Triple Negative Breast Cancer Revealed by Multiplexed Ion Beam Imaging.

The immune system is critical in modulating cancer progression, but knowledge of immune composition, phenotype, and interactions with tumor is limited. We used multiplexed ion beam imaging by time-of-flight (MIBI-TOF) to simultaneously quantify in situ expression of 36 proteins covering identity, function, and immune regulation at sub-cellular resolution in 41 triple-negative breast cancer patients. Multi-step processing, including deep-learning-based segmentation, revealed variability in the composition of tumor-immune populations across individuals, reconciled by overall immune infiltration and enriched co-occurrence of immune subpopulations and checkpoint expression. Spatial enrichment analysis showed immune mixed and compartmentalized tumors, coinciding with expression of PD1, PD-L1, and IDO in a cell-type- and location-specific manner. Ordered immune structures along the tumor-immune border were associated with compartmentalization and linked to survival. These data demonstrate organization in the tumor-immune microenvironment that is structured in cellular composition, spatial arrangement, and regulatory-protein expression and provide a framework to apply multiplexed imaging to immune oncology.

Measuring Signaling and RNA-Seq in the Same Cell Links Gene Expression to Dynamic Patterns of NF-κB Activation.

Signaling proteins display remarkable cell-to-cell heterogeneity in their dynamic responses to stimuli, but the consequences of this heterogeneity remain largely unknown. For instance, the contribution of the dynamics of the innate immune transcription factor nuclear factor κB (NF-κB) to gene expression output is disputed. Here we explore these questions by integrating live-cell imaging approaches with single-cell sequencing technologies. We used this approach to measure both the dynamics of lipopolysaccharide-induced NF-κB activation and the global transcriptional response in the same individual cell. Our results identify multiple, distinct cytokine expression patterns that are correlated with NF-κB activation dynamics, establishing a functional role for NF-κB dynamics in determining cellular phenotypes. Applications of this approach to other model systems and single-cell sequencing technologies have significant potential for discovery, as it is now possible to trace cellular behavior from the initial stimulus, through the signaling pathways, down to genome-wide changes in gene expression, all inside of a single cell.

Deep Learning Automates the Quantitative Analysis of Individual Cells in Live-Cell Imaging Experiments.

Live-cell imaging has opened an exciting window into the role cellular heterogeneity plays in dynamic, living systems. A major critical challenge for this class of experiments is the problem of image segmentation, or determining which parts of a microscope image correspond to which individual cells. Current approaches require many hours of manual curation and depend on approaches that are difficult to share between labs. They are also unable to robustly segment the cytoplasms of mammalian cells. Here, we show that deep convolutional neural networks, a supervised machine learning method, can solve this challenge for multiple cell types across the domains of life. We demonstrate that this approach can robustly segment fluorescent images of cell nuclei as well as phase images of the cytoplasms of individual bacterial and mammalian cells from phase contrast images without the need for a fluorescent cytoplasmic marker. These networks also enable the simultaneous segmentation and identification of different mammalian cell types grown in co-culture. A quantitative comparison with prior methods demonstrates that convolutional neural networks have improved accuracy and lead to a significant reduction in curation time. We relay our experience in designing and optimizing deep convolutional neural networks for this task and outline several design rules that we found led to robust performance. We conclude that deep convolutional neural networks are an accurate method that require less curation time, are generalizable to a multiplicity of cell types, from bacteria to mammalian cells, and expand live-cell imaging capabilities to include multi-cell type systems.

A single-molecule Hershey-Chase experiment.

Ever since Hershey and Chase used phages to establish DNA as the carrier of genetic information in 1952, the precise mechanisms of phage DNA translocation have been a mystery. Although bulk measurements have set a timescale for in vivo DNA translocation during bacteriophage infection, measurements of DNA ejection by single bacteriophages have only been made in vitro. Here, we present direct visualization of single bacteriophages infecting individual Escherichia coli cells. For bacteriophage λ, we establish a mean ejection time of roughly 5 min with significant cell-to-cell variability, including pausing events. In contrast, corresponding in vitro single-molecule ejections are more uniform and finish within 10 s. Our data reveal that when plotted against the amount of DNA ejected, the velocity of ejection for two different genome lengths collapses onto a single curve. This suggests that in vivo ejections are controlled by the amount of DNA ejected. In contrast, in vitro DNA ejections are governed by the amount of DNA left inside the capsid. This analysis provides evidence against a purely intrastrand repulsion-based mechanism and suggests that cell-internal processes dominate. This provides a picture of the early stages of phage infection and sheds light on the problem of polymer translocation.

Ion-dependent dynamics of DNA ejections for bacteriophage lambda.

We studied the control parameters that govern the dynamics of in vitro DNA ejection in bacteriophage lambda. Previous work demonstrated that bacteriophage DNA is highly pressurized, and this pressure has been hypothesized to help drive DNA ejection. Ions influence this process by screening charges on DNA; however, a systematic variation of salt concentrations to explore these effects has not been undertaken. To study the nature of the forces driving DNA ejection, we performed in vitro measurements of DNA ejection in bulk and at the single-phage level. We present measurements on the dynamics of ejection and on the self-repulsion force driving ejection. We examine the role of ion concentration and identity in both measurements, and show that the charge of counterions is an important control parameter. These measurements show that the mobility of ejecting DNA is independent of ionic concentrations for a given amount of DNA in the capsid. We also present evidence that phage DNA forms loops during ejection, and confirm that this effect occurs using optical tweezers.

Biochemistry on a leash: the roles of tether length and geometry in signal integration proteins.

We use statistical mechanics and simple ideas from polymer physics to develop a quantitative model of proteins whose activity is controlled by flexibly tethered ligands and receptors. We predict how the properties of tethers influence the function of these proteins and demonstrate how their tether length dependence can be exploited to construct proteins whose integration of multiple signals can be tuned. One case study to which we apply these ideas is that of the Wiskott-Aldrich Syndrome Proteins as activators of actin polymerization. More generally, tethered ligands competing with those free in solution are common phenomena in biology, making this an important specific example of a widespread biological idea.